Repository navigation
[Xamarin.Android.Build.Tasks] Generate R8 JNI remapping tables - #12848
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The review found build-blocking issues (missing namespace import for required extension methods and nullable-unsafe XML attribute reads) that should be fixed before merge.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (3)
What changed in this PR
This PR adds the producer-side implementation for R8 JNI runtime remapping in Xamarin.Android.Build.Tasks. It introduces parsing/enumeration of R8 mappings, retention filtering for linked managed assemblies and NativeAOT post-ILC objects, and generation of native lookup tables (LLVM IR) including forward/reverse type maps and method/field indices—without rewriting managed assemblies.
Changes:
- Add
GenerateR8JniRemappingtask plus supporting utilities to parse R8 mappings, filter required entries, and emit runtime remapping XML. - Add managed metadata and NativeAOT object scanning to retain only mappings that still have consumers.
- Replace the prior remapping LLVM generator with a new generator that also supports reverse type tables and field replacement index tables, plus new/updated test coverage and resource strings.
| File | Description |
|---|---|
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemappingNativeCodeGenerator.cs | New LLVM IR generator emitting type/method/field remapping tables and counts. |
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemappingAssemblyGenerator.cs | Removed older generator (superseded by native-code generator). |
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemapping/R8Mapping.cs | Adds field type capture and deterministic enumeration APIs for retention/codegen. |
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemapping/NativeAotJniRetention.cs | New NativeAOT object scan + literal matcher for post-ILC retention selection. |
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemapping/JniRemappingAssemblyScanner.cs | New PE/metadata scanner to retain only referenced mappings from linked assemblies. |
| src/Xamarin.Android.Build.Tasks/Utilities/JniRemapping/JniDescriptorText.cs | New descriptor parsing/validation/conversion utilities. |
| src/Xamarin.Android.Build.Tasks/Tasks/GenerateR8JniRemapping.cs | New task converting R8 mapping to remapping XML with conflict handling. |
| src/Xamarin.Android.Build.Tasks/Tasks/GenerateJniRemappingNativeCode.cs | Extended XML parsing and table generation to include reverse types + fields. |
| src/Xamarin.Android.Build.Tasks/Properties/Resources.resx | Adds XA4327/XA4328 strings for new error/warning reporting. |
| src/Xamarin.Android.Build.Tasks/Properties/Resources.Designer.cs | Designer updates for new resource entries. |
| src/Xamarin.Android.Build.Tasks/Tests/.../R8MappingTests.cs | New tests for deterministic enumeration + method-key splitting. |
| src/Xamarin.Android.Build.Tasks/Tests/.../JniRemappingAssemblyScannerTests.cs | New tests validating metadata-driven retention and proxy behavior. |
| src/Xamarin.Android.Build.Tasks/Tests/.../JniDescriptorTextTests.cs | New tests for descriptor rewrite/validation/conversion. |
| src/Xamarin.Android.Build.Tasks/Tests/.../GenerateR8JniRemappingTests.cs | New task-level tests including NativeAOT retention filtering. |
| src/Xamarin.Android.Build.Tasks/Tests/.../GenerateJniRemappingNativeCodeTests.cs | New tests covering empty tables, reverse/field tables, and LLVM validity. |
Files not reviewed (1)
- src/Xamarin.Android.Build.Tasks/Properties/Resources.Designer.cs: Generated file
ff61e44 to
3c36a33
Compare
## Summary Remove the experimental build-time managed-assembly rewriting approach for R8 so any future assembly-rewriting design can start from a clean foundation. This cleanup is related to #12535 and the original prototype in #12575. ## What this PR undoes This removes the managed rewriting implementation developed across the R8 obfuscation stack: - #12629 added the PE metadata rebuild substrate. - #12630 added managed JNI metadata rewriting from R8 mappings. - #12631 added rewriting for generated trimmable type-map assemblies. - #12632 and #12634 integrated and tested that rewriting approach for CoreCLR and NativeAOT. Concretely, this PR removes the rewrite task, rewrite-only metadata/IL utilities, dedicated tests and fixtures, and the XA4325/XA4326 resources and documentation. The intent is to abandon this implementation rather than preserve an unused rewriting stack that a future design would need to work around. ## What remains in place - The generic `R8Mapping` parser introduced in #12628 remains, along with its tests and the `MSBuildDeviceIntegration` consumer. It is independently useful for reading R8 mapping files. - The ordinary R8 configuration and `private-members` obfuscation/optimization policy from #12668 remain unchanged. This PR does **not** disable R8 or remove private-member obfuscation. - Existing D8/R8 packaging, ProGuard rule handling, and non-rewriting build behavior remain unchanged. - Runtime remapping remains active as the stacked follow-up work in #12847, #12848, #12692, and #12844. Those PRs implement the alternative opt-in strategy without managed assembly rewriting and are not part of this cleanup diff. We may revisit managed assembly rewriting in .NET 12 based on customer feedback and performance data, but with a fresh design rather than this implementation. ## Validation - Built `Xamarin.Android.Build.Tasks` - Ran `Microsoft.Android.Build.Tasks.Tests` - Ran `Microsoft.Android.Sdk.TrimmableTypeMap.IntegrationTests` - Built `MSBuildDeviceIntegration` - Ran focused `R8MappingTests`
9b96151 to
1c092d6
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Port the managed remapping data ABI from the closed runtime stack, remove the native lookup implementations and P/Invokes, and retain the extended reverse-type, field, signature, and declaring-owner behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Dispose cached instance-method redirects and cover the managed generated-table lookup paths for Unicode ordering and member specificity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Dispose remapped member and subclass-constructor candidates that lose ConcurrentDictionary publication races, and document the single-hop type-remapping contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Forward field remapping through AndroidTypeManager and use an XML-safe high Unicode key for the lookup boundary test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto review |
dalexsoto
left a comment
There was a problem hiding this comment.
The full 18-file review, separate producer-to-runtime integration/completeness pass and final independent sweep identify seven current correctness blockers. The producer-only scope, optional target-signature fallback and already-fixed namespace/signature/IL-scanning findings are accepted; prerequisite #12847 is present.
-
Recognize the linked framework implementation identities (JniRemappingAssemblyScanner.cs:151-184). ILLink at the pinned dependency resolves forwarded
TypeMapAttribute<T>and constructorSystem.Typereferences toSystem.Private.CoreLib. The facade-only checks silently discard valid surviving entries; repairing only the attribute check still rejects the constructor. Generated proxies have no ordinary registration/removed-IL fallback. Support genuine facade and implementation identities for both checks, preserve lookalike rejection, and add implementation-scoped metadata coverage. -
Select the Java TypeMap universe before JNI validation (scanner:160-161). The generic group argument is discarded. Actual
TypeMap<JavaDictionary>entries inValueTypeDictionaryFactoryuse CLR keys such asSystem.Collections.Generic.IDictionary2[System.Boolean,System.Boolean]; interpreting these as JNI class names raisesBadImageFormatException`/XA4325 and prevents output. Skip unrelated framework mapping groups before parsing their keys; fixing implementation-scope recognition also exposes this valid linked input. -
Retain class-only surviving entries explicitly (scanner:286-304).
RecordAllMappingsrecords only field/method accesses. A renamed marker interface/class with no enumerated member mappings adds noCentry, soGenerateContentremoves both required forward and reverse type mappings despite its surviving authoritative TypeMap key. Mark class access independently and add a class-only retention assertion. -
Filter JNI method attributes before decoding (scanner:315-332). Every method attribute is decoded before the name switch. The reused dummy provider cannot decode enum/System.Type arguments, so a valid unrelated attribute such as
EditorBrowsable(Never)becomes a false malformed-assembly XA4325 error andRunTaskexits before writing output. Determine a supported JNI attribute name before decoding, as the type/field paths do, and cover unrelated valid attributes. -
Preserve descriptor-distinct field rows (R8Mapping.cs:141-145). Name-only dictionaries overwrite
int value -> awithjava.lang.String value -> b, a valid descriptor-distinct JVM layout. The new typed enumeration emits only the latter, while the actual field lookup requires name and source descriptor; the integer consumer loses its remap. Preserve name-plus-type/descriptor identity and all rows, with name-based metadata retention retaining every required variant. This is a tightly coupled new producer failure, not a blanket complaint about the old name-only API. -
Do not turn a longer NativeAOT class literal into phantom class retention (NativeAotJniRetention.cs:20-25,213-228). Unbounded matching retains
com/contoso/Peerinside a genuinely retainedcom/contoso/Peer$Innerliteral. If R8 maps those original classes to one residual class, the phantom parent makes the reverse map falsely ambiguous and suppresses the entry needed by the surviving inner class. Use authoritative typemap keys or complete-literal/descriptor-token boundaries; preserve the intentional policy for truly ambiguous merges and add the prefix/merged-class case. -
Claim ownership only for XML the merger accepts (GenerateR8JniRemapping.cs:375-384). A well-formed
<ignored><replace-type from="com/contoso/Peer" to="a/b" /></ignored>is scanned as an existing owner, suppressing its identical generated forward mapping.MergeRemapXmlthen rejects the entire input because the root is notreplacements, leaving the required mapping absent. Validate existing documents against the merger's acceptance rules before committing their ownership claims and cover rejected-root input.
All seven map to this exact head and were consolidated after every changed file/material path/candidate and the final sweep were accounted for. These are source/metadata/consumer proofs, not executed repository, Java, R8 or native tests; current successful checks do not exercise the counterexamples.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto Addressed the seven blockers in three follow-up commits:
Validation:
This remains producer-only: no managed assembly rewriting, runtime table ABI changes, product policy, or pipeline wiring. Requesting another review; the consolidated changes-requested review has no individual unresolved inline threads to resolve. |
Validate a pooled, immutable snapshot of each XML input before emitting its mappings. Reject malformed documents entirely with XA4318, preventing conflicting leading entries from leaking into generated remap output. Preserve adjacent elements without advancing the reader twice after WriteNode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove descriptor entry points used only by tests and derive external type ownership from existing entry keys. Consolidate scanner fixtures and overlapping XML/field tests while retaining linked-metadata, LLVM, transactional XML and NativeAOT regressions. Strengthen lookalike attribute coverage using a recognized Java universe for both local and external user identities. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
dalexsoto
left a comment
There was a problem hiding this comment.
The full 20-file/current-hunk review, separate producer-to-runtime integration pass and final independent sweep confirm the framework identities, universe filtering, class-only retention, unrelated-attribute filtering, typed fields and XML ownership corrections. Phantom NativeAOT retention is only partially fixed; one UTF-16 edge-identity blocker remains.
Preserve complete UTF-16 code units at trimmed edges (NativeAotJniRetention.cs:406-423). For mappings com.contoso.Peer -> a.b and com.contoso.PeeŲ -> a.b, a native literal containing only the latter still retains Peer. Peer ends in 72 00; PeeŲ ends in 72 01. The shortened pattern matches through 72 although the complete Peer encoding is absent. Because the completing 01 is nonzero, the boundary code does not restore/validate the omitted zero and reads 01 00 as the next character, U+0001, which passes the continuation check.
The phantom class makes a/b falsely ambiguous, suppressing the reverse entry needed by the surviving peer; descriptor-identical members with different residual targets can instead cause a false XA4325 conflict. Structural exclusion of dehydration command bytes does not fix this: independently modeled tagged ELF32/ELF64 literal regions retain the same mismatch. A leading-edge counterpart also matches ĀPeer inside ĠPeer.
Account only for zeros genuinely supplied/elided by dehydration while preserving the full code-unit identity; reject nonzero completing bytes and add leading/trailing Unicode identity regressions asserting retained classes and reverse output. This is the residual reported phantom-retention concern, not an objection to conservative handling of genuinely ambiguous merges or accepted producer-only staging. Proof is immutable source and standalone RAM byte/format calculations, not executed ILC, C#, Java or R8 tests. No additional high-confidence blocker remained.
…ntity Match complete UTF-16 code units instead of accepting shortened leading/trailing zero-byte patterns. Reconstruct only zeros explicitly encoded by ZeroFill; relocation and stream boundaries no longer supply invented bytes. Keep short zero runs within literal regions and bound long padding without changing genuine merged-class ambiguity policy. Reproduce both Unicode edge aliases in raw and tagged ELF32/ELF64 inputs, verify forward/reverse tables and absence of phantom member conflicts, and reject code units truncated at unknown relocations. Replace untagged dehydration fixtures with actual Copy/ZeroFill command streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto Fixed the remaining UTF-16 edge-identity blocker in The matcher now uses complete UTF-16 byte sequences for both classes and members—no shortened leading/trailing-zero patterns. The dehydration decoder restores only bytes explicitly supplied by The new regression reproduced both The Xamarin.Android.Build.Tasks/test-project build passed, and all 130 focused remapping tests passed with zero skips, including actual ILLink forwarding and LLVM/llc checks. Final read-only review completed. Normal follow-up push only; PR base and stack metadata are unchanged. Please re-review the new head. |
dalexsoto
left a comment
There was a problem hiding this comment.
The previously reported UTF-16 edge-trimming cases are fixed. The complete current 20-file/36-hunk review, separate producer-to-runtime integration pass and final independent sweep leave one different class-evidence blocker.
Match actual string payloads and encodings (NativeAotJniRetention.cs:374-409). Both UTF-8 and UTF-16 patterns scan untyped buffers. A frozen string containing com/contoso/ plus53 B characters has length65: its copied Int32 header 41 00 00 00 retains absent class A inbothencodings. Independently, genuine UTF-16 classU+0141 (41 01) retains absent A viaitsUTF-8pattern 41,because the following controlbyte 01 passes the boundary check. Neither case requires shortened patterns or inventedzeros.
These phantom classes falsely make a shared R8 residual name ambiguous, suppressing the reverse mapping required by the actual surviving peer; descriptor-identical members with different residual targets can instead cause false XA4325 conflicts and prevent output. Restrict class evidence to identified literal payloads and actual encodings, excluding headers/executable bytes, and cover exact-survivor/reverse/member-conflict cases. Conservative member matching, genuine merged-class ambiguity and producer-only scope remain accepted.
This route was already discoverable on the earlier reviewed afecd5e/3ad9a4 heads and was a review-completeness miss, not introduced by your UTF-16 correction. Proof is independently verified current/pinned ILC source and static RAM byte calculations, not executed repository/compiler/device tests. The failed external MAUI check stopped at APK installation with INSTALL_FAILED_INSUFFICIENT_STORAGE/invalid UID; it is not presented as an additional R8 source blocker or a green-CI claim.
Restrict class retention to named, length-delimited frozen UTF-16 strings, explicitly marked ELF UTF-8 string sections, and Java-universe NativeFormat typemap keys. Resolve ELF32/ELF64 hydration destinations and common fixups structurally; track sparse Copy/ZeroFill ranges without interpreting headers, executable bytes, relocation bytes, or another encoding as class evidence. Keep member matching conservative and fail closed on incomplete frozen payloads. Reuse the existing NativeFormat parser and shared typemap group/name validation rather than maintaining duplicate parsers. Cover the 65-character length-header and U+0141 alias counterexamples, exact reverse mappings/member conflicts, untyped/executable data, raw/tagged encodings, native Java vs CLR universes, mixed rodata, and empty strings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto Addressed the remaining class-evidence blocker in Class retention no longer scans arbitrary object/dehydration buffers. It uses:
For dehydrated strings, ELF32 REL/ELF64 RELA identify the hydration destination. Sparse Copy/ZeroFill ranges preserve virtual offsets and known bytes without treating relocations as characters. Each class pattern must match the payload’s actual encoding; UTF-16 matches must also start on a code-unit boundary. Member matching and genuinely ambiguous merged-class policy remain conservative and unchanged. The existing NativeFormat parser and group/name validators are source-shared with the modern tasks rather than duplicated. The original counterexample regression failed for both ELF32/ELF64 before the fix. The final regressions isolate the 65-character Validation passed: the classic task/test-project build; all 142 focused remapping tests (zero skips), including real ILLink and LLVM/llc checks; and all 67 modern task tests after sharing the helpers. Final read-only review completed. No device/full SDK validation or actual ILC compilation was run for this follow-up. Normal push only; PR base and stack metadata unchanged. Please re-review the new head. |
dalexsoto
left a comment
There was a problem hiding this comment.
The previous length-header and cross-encoding class-evidence cases are fixed, and the producer-only scope, conservative member matching and genuine merged-class ambiguity remain accepted. The complete current 25-file/44-hunk review and separate integration/final sweep leave one new valid-input blocker.
Identify actual frozen-string symbols, not any __Str_ substring (NativeAotJniRetention.cs:196-205). A legitimate retained managed type such as N.__Str_Helper in assembly App has an ordinary MethodTable symbol _ZTV18App_N___Str_Helper, which now enters the FrozenStringNode reader. With normal ILC data dehydration, its EEType is misread using string-object offsets: on ELF64 the attempted length at offset 8 falls inside the related-type relocation; on ELF32 BaseSize at offset 4 becomes the string length and the payload starts at that same relocation. The sparse reader correctly rejects the unknown bytes, but the false classification causes XA4325 and prevents remapping XML even when this unrelated type has no JNI mappings.
Constrain classification to actual compiler-frozen string identities/regions while preserving fail-closed handling of genuinely incomplete strings, and cover ordinary non-string marker collisions in both ELF classes. This is introduced by the current provenance replacement, not the older header/UTF16 completeness miss. Proof is independently verified dependency-pinned compiler/ELF source and RAM-only layout reasoning, not executed ILC, repository or device tests; no additional independent blocker remained.
Require the compilation-unit-specific frozen string prefix and membership in the corresponding __FrozenSegmentStart region before reading a string. Bound its header and payload by that region so ordinary managed MethodTable names containing __Str_ are ignored, while incomplete actual frozen strings continue to fail closed. Prefer decoded hydration ranges over file-backed zero placeholders, matching the ILC dehydration emission. Reproduce the managed marker collision on ELF32/ELF64 and cover BSS/file-backed hydration, same-prefix symbols outside the frozen region, and incomplete actual strings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto Addressed the frozen-symbol collision in The reader now derives each compiler string prefix from the corresponding The new fixture models Source inspection of The task/test-project build passed and all 148 focused remapping tests passed with zero skips, including real ILLink and LLVM/llc checks. Final read-only review completed. No actual ILC compilation or device/full SDK validation was run for this follow-up. Normal push only; PR base and stack metadata unchanged. Please re-review the new head. |
dalexsoto
left a comment
There was a problem hiding this comment.
The ordinary MethodTable/frozen-string collision and hydration-source ordering are fixed. The complete current 25-file/44-hunk review, separate integration pass and final independent sweep leave two correctness blockers:
-
Distinguish primitive descriptor tokens from class references (NativeAotJniRetention.cs:652-663). A genuine frozen member string
run.(I)Vretains absent default-package classI: its complete, aligned UTF-16 pattern is bounded by(and), which the bare-class matcher accepts. WithI -> a.bandcom.contoso.Peer -> a.b, but only Peer actually retained, this phantom class suppresses the required Peer reverse mapping. Different residualrun(int)targets also cause a false XA4325 conflict before output. Make bare-class evidence descriptor-aware while preserving genuineLI;object references, and cover exact-survivor/reverse/conflict cases on raw/dehydrated ELF32 and ELF64. -
Recognize the pinned compiler's disambiguated Java-group identities (TypeMapKey.cs:35-38). A valid application/reference named
Mono-Androidsorts beforeMono.Android; both sanitize toMono_Android, so NativeAotNameMangler names the framework assemblyMono_Android_0. Its real Java group relocation is_ZTV31Mono_Android_0_Java_Lang_Object, which the fixed-spelling predicate rejects. A disambiguated local TypeMap assembly similarly loses the required anchor suffix.ReadKeysthen skips the genuine group, silently dropping key-only surviving classes and their remapping rows. Account for actual mangler disambiguation without admitting CLR universes, and cover shared/local-anchor collision shapes.
Both cases were already discoverable on the previously reviewed 73e244b head and were misses in my earlier review, not regressions introduced by this correction. The evidence is independently verified current source, dependency-pinned compiler/ELF contracts and RAM-only byte/name calculations, not executed repository, ILC or device tests. Producer-only scope, conservative member evidence and genuine merged-class ambiguity remain accepted. Current CI is not claimed green; its observed NativeAOT Dispose_Finalized failure is not asserted flaky or an additional proven R8 source defect.
…tities Require bare class names to occupy the complete identified literal. Primitive tokens inside method/array descriptors are not independent classes; full object descriptors such as LI; continue to retain genuine class references. Preserve conservative member evidence and genuine merged-class ambiguity. Recognize NativeAotNameMangler canonical numeric disambiguation suffixes on shared Java assembly and local typemap anchor identities. Validate MethodTable symbol framing and retain CLR/lookalike universe exclusions in the shared predicate. Reproduce false primitive member conflicts and key-only group loss before the fixes; cover raw/dehydrated ELF32/ELF64, real object references, collision suffixes and negative groups. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@dalexsoto Addressed both latest findings in
Validation passed: the task/test-project build; all 166 focused producer/remapping tests with zero skips (including actual ILLink forwarding and LLVM/llc checks); and all 78 modern task tests. Final read-only review completed. These are executed repository fixtures plus compiler-source contract verification, not an actual ILC/device/full SDK run. I am not claiming CI is green or classifying the observed Dispose_Finalized failure as flaky. Normal follow-up push only; PR base and stack metadata are unchanged. Please re-review the new head. |
dalexsoto
left a comment
There was a problem hiding this comment.
The complete 26-file re-review confirms both remaining fixes. Whole-literal class matching excludes primitive descriptor tokens while retaining genuine object references, and the shared canonical suffix/framing predicate agrees with the pinned compiler in both native consumers. The previously fixed metadata, provenance, typed-field, XML ownership and native lookup contracts remain preserved. No blocking issue remains within the accepted producer-only scope.
A surviving Java TypeMap attribute establishes class identity, not that every R8-mapped member still has a managed consumer. Recording all members restores remapping entries for methods and fields absent from final linked metadata. Record only the class mapping and leave member retention to surviving JNI metadata. Remove the unused all-member helper. Cover class-only and mixed fixtures, both attribute forms, forward/reverse XML, field descriptor variants, and byte-for-byte input assembly immutability. Context: #12848 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Depends on #12847.
This layer adds the producer-side implementation for R8 JNI runtime remapping. It parses R8 mappings, scans linked managed metadata read-only to retain surviving entries, supports NativeAOT post-ILC retention, and generates forward/reverse type, method, field, descriptor, indexed, count, and empty LLVM lookup tables.
Managed assemblies are never rewritten or reconstructed by this change.
This intentionally contains no public product mode, policy, pipeline target wiring, NativeAOT deferred-link orchestration, documentation, or end-to-end device tests; that wiring remains for the later #12692 layer.